A SOUND CONTROL INSTALLATION 



Field of the invention 

[0001] The present invention relates to an installation to control by sounds electrical 
units. 

[0002] It is known to control by sounds, for example in home applications, electrical 
units such as lamps, plugs... Using sounds (for example hand clicks) to control electrical 
units is particularly convenient as it does not require the user to physically act on a switch 
or a remote control apparatus. 

[0003] However, providing a space such as a room with sound controllable units 
poses several problems. 

[0004] A first problem is to distinguish a sound order from an accidental noise (for 
example, a fall of an object on the floor). 

[0005] Another problem appears when more than one user is present in the room. 
Then, a conventional sound controllable unit cannot identify a user from another one. 

[0006] Another problem is that the conventional equipments are not adapted to allow 
controlling more than one unit in a same room. In particular, providing a single room 
with two sound controllable units needs coding the sound control messages. This is 
complex and increases the number of environmental noises, which may be considered as 
parasitic control orders. 

[0007] The present invention aims at providing a sound control installation for 
controlling electrical units which overcomes the drawbacks of the known equipments. 

[0008] Another purpose of the present invention is to provide an installation which 
does not require the user to physically act on a control element. 

[0009] Another purpose of the present invention is to allow controlling, in a same 
room, several electrical units without needing to individualize the control sound. 
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{0010] Another purpose of the invention is to distinguish sound orders coming from 
different users in a same room. 

[0011) To attain these purposes and others, the present invention provides a remote 
control device capable of communicating with electrical units to be controlled by means 
of wired or wireless links, only the control device being controllable with sound by a 
user. 

[0012] According to the present invention a schematic 3D-view of a room with the 
respective locations of electrical units to be controlled is displayed on a screen and, in a 
predetermined perimeter or area of the room, a hand of a user is tracked with stereo 
cameras and is used to displace a cursor on the screen. A 3D-microphone array is also 
provided in order to identify the origin of a sound. 

[0013] Alternatively, a third camera on the opposite side of the room is used to take 
pictures of the real units to be controlled and/or for updating in real time the pictures of 
the control screen. 

[0014] The tracking of a hand in the area covered by the cameras uses a conventional 
shape recognition system in video pictures. Further, using a hand of a user as a "mouse" 
for pointing a cursor of a computerized screen is also known. For example, the Sony 
VAIO PCG-C1XS system has a piece of software called "Smart Capture Application" 
and a camera called "Motion Eye" which together have the ability to capture the index 
finger of the user and link its movement to the movement of the mouse arrow on the 
computer screen. 

[0015] According to a preferred embodiment of the present invention, the installation 
is turned on by a sound which is sensed by the microphones. Such an embodiment 
allows, as it will be better understood later, to distinguish several hands which could be 
present in the area covered by the cameras. The selected hand will be the closest from 
the location at which the sound has been detected. 
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(0016] The system can also check the pointed unit on the control screen by 
announcing through a loud-speaker an identifiant of the pointed pictogram when the 
cursor comes on that pictogram. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0017] These purposes, features and advantages of the invention will become 
apparent from the following detailed description of embodiments, given by way of 
illustration and not limitation with reference to the accompanying drawings. 

[0018] Figure 1 represents very schematically a room provided with an installation 
according to the present invention; and 

[0019] Figure 2 illustrates a control screen used in an installation for controlling the 
room of figure 1 . 

[0020] For clarity, only the elements useful to the understanding of the invention 
have been shown in the drawings and will be disclosed hereafter. Especially, the 
programming steps which have to be made in order to implement the installation 
according to the present invention will not be detailed as they will readily occur to those 
skilled in the art. Further, known equipment for determining the location of a hand 
considered as a cursor, used in the present invention for the control screen, will not be 
disclosed as known. 

[0021] Detailed description of the invention 

[0022] Figure 1 represents a room 1 provided with an installation for sound 
controlling electrical elements according to the invention. 

[0023] The control installation of the invention comprises a control device 2, a 
control screen 3 and image and sound sensors. In the represented embodiment, two video 
cameras 4 and 5 are disposed on the corner of the screen 3 and three microphones 6, 7 
and 8 are also disposed around the screen. Each microphone and camera is linked to the 
control device 2 which controls the screen 3. 
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[0024] The sensors of the same type (video or audio) are not located at a same 
position so as to be capable of localizing the sound and picture sources. 

[0025] The microphones can be located anywhere in room 1 and linked (wire or 
wireless) to device 2. The video cameras 4 and 5 are oriented to watch an area A located 
in front of the screen 3 so as to film the hand H of a user U who wants to control the 
electrical units using sound. 

[0026] The electrical units are, for example, a ceiling light 10, two bracket lamps 1 1 
and 12, two wall sockets 13 and 14, and a switch 15. These electrical units are 
distributed in the room and are linked to the control device 2. Each electrical unit 
comprises a radiofrequency receiver R10, Rl 1, R12, R13, R14 and R15 communicating 
with the control device 2 to receive control orders. Alternatively, the electrical units to 
be controlled by the installation according to the invention can be wire connected to the 
control device 2. 

[0027] An important feature of the invention is that the electrical units that will be 
rendered sound controllable by the present invention are not individually sound 
controllable. 

[0028] Screen 3 is not necessarily contiguous with the control device 2 provided that 
it is linked to this device and it is visible from the area A watched by cameras 4 and 5, 
and within line of sight from the microphones. For example, screen 3 can be the screen 
of a TV set equipped to be controlled by device 2. Then, the area A covered by the 
cameras 4 and 5 is preferably the area from which the users might be watching TV, for 
example an area around a sofa 20 disposed in front of the screen 3. 

[0029] Figure 2 represents an image on screen 3 when the installation is on. 
According to the invention, the representation is preferably a perspective to display not 
only the wall Wl of the screen but also the floor Fl, the roof ROl, and the walls WR1 
and WL1, respectively right and left to the wall Wl. 

[0030] On the screen, the control device 2 displays not only the shape of the room 1 
but also, according to a preferred embodiment, pictograms P10, P11,P12, P13,P14 and 
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P15 respectively figuring the units 11,12, 13, 14 and 15 to be controlled by the 
installation. 

[0031] The generation of pictograms P is obtained in a configuring phase of the 
software controlling the control device 2. 

[0032] According to a first variant, the user (or the installer) defines the walls of the 
room and the locations of the pictograms using a conventional graphic software. 

[0033] According to a second variant, the installation automatically acquires the 
controllable units at each activation of the installation. According to that embodiment, a 
third camera (not shown) is provided to film the wall of the room containing the screen 
arid to be able to locate all elements in the room. Manual registration could be included 
as the control space is 3D, and hence easily definable through a coordinate representation 
on a computer. The selection of the elements of the room which have to be displayed on 
the control screen 3 as pictograms is then made by using the communication links 
between the controllable units and the control device 2. For example, the elements R10 
to Rl 5 are not only radiofrequency receivers but also radiofrequency emitters. Then, 
when the installation is turned on, a request message is sent to all the possibly connected 
electrical units. The respective units respond with an identifier to allow the identification 
of the different units. If necessary, the transmission between the various units and the 
control device 2 is also used to assist the localization made by the video system. The 
identification of the various electrical units can be used to automatically select a 
pictogram (socket, switch, lamp, etc.) chosen in a library of the installation. 

[0034] According to a third variant, the representation of the control screen 3 is a real 
representation. Then, the transceivers of the controllable units are only used to locate on 
the picture the area in which the cursor has to be considered as selecting a unit. 

[0035] The operation of the installation is for example as follows. The cameras 4 and 
5 permanently monitor the area A (figure 1) and the images are processed to identify the 
presence of a hand. Known detection systems of human shapes like hands usually use 
colour differentiation to more quickly isolate the skin area on a picture. The detection of 
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the position of a hand in a dedicated area is made by conventional techniques. If needed, 
a reference object can be disposed in the field of the cameras to help in matching the 
referentials of the images of the cameras. 

(0036] Once the hand H of a user U (figure 1 ) has been detected, the system 
calculates the displacements of the hand between two successive video pictures to 
transfer this movement on the cursor C displayed on the control screen 3. Then, the user 
can see the cursor C move along with his hand displacement to select a unit to control. 

[0037] Preferably, once the cursor C encounters on the screen a pictogram P of a unit, 
the user is made aware that a unit can be selected. For example, the system can announce 
through speakers the name and type of the unit selected. Alternatively, the corresponding 
pictogram can be highlighted on the screen. 

[0038] Having selected a controllable unit, the user can make a click with his hand, or 
produce another noise, to control the corresponding unit. This sound is sensed by the 
three microphones 6, 7 and 8 and processed to check that it originates from the hand H, 
or its close neighbourhood. For this purpose, the control device 2 calculates the 
difference between the times of arrival of the sound on the different microphones. 
Knowing the location of the microphones, the system is then capable of calculating the 
2D or 3D location of the sound source. 

[0039] If the origin of the sound substantially corresponds to the location of the hand 
H detected by the cameras, then the installation executes the appropriate control. 

[0040] If not, the installation ignores the sound order which is to be considered as a 
parasitic noise. 

[0041] An advantage of the present invention is that the installation is able to 
distinguish perturbating noise from a control order. 

[0042] Another advantage of the system of the present invention is that it is possible 
to individually control more than one unit is the same room. 
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[0043] To distinguish the hands of a user and select only one hand for controlling the 
cursor C, various solutions can be adopted. A first solution is to consider the hand 
controlling the cursor as that with the highest degree of motion. Then, the user wanting 
to use the system knows that he has to move one hand in the area A more than the other 
and that this hand will be used to control the cursor. Further, the user will see that the 
cursor displayed on screen 3 follows the displacement of his hand. Another solution is to 
identify the position (open or closed) of the hands and, preferably, the existence of a 
finger pointed in the alignment of the arm of the user and to select this shape for 
controlling the cursor on the screen. The implementation of that solution only needs the 
control device 2 to be equipped to detect shapes like arms and hands. 

[0044] According to a preferred embodiment of the present invention, the selection of 
the hand of a user which has to control the position of the cursor C is made at the 
activation of the installation, and the installation is sound activated. In other words, the 
three microphones 6, 7 and 8 permanently listen to the noise in the room, the cameras 4 
and 5 being off. When detecting a sound which may be considered as an hand click, the 
control device 2 switches on cameras 4 and 5 and the first pictures taken by the cameras 
are analyzed in an area corresponding to the origin of the sound calculated by 
triangulation with microphones 6, 7 and 8. If a shape recognized as a hand is located in 
this area, the installation considers that this hand has to be tracked to control the cursor 
location on the screen. If no hand shape is detected in this area, the installation considers 
that the noise is a parasitic one. Alternatively, both cameras and microphones can 
monitor the room permanently. 

[0045] Also, one could also provide two or more cursors on the screen corresponding 
to more than one hand. Then, all the cursors will be followed by the camera and the 
selection of the cursor to be taken into account when a sound is produced can be made by 
controlling the origin of the sound. 

[0046] The practical implementation of the present invention is in the ability of one 
ordinary skilled in the art in view of the functional explanations above. In particular, the 
programming of a software to be used for initializing and operating the installation 
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according to the invention is not to be detailed as it is in the ability of those skilled in the 
art. 

[0047] Even if the present invention has been disclosed in connection with a 
particular application to home environment, the invention more generally applies to any 
environment or space where a sound control installation can be used for controlling more 
than one unit. For example, the area surveyed by the installation can be a car, a storage 
house... 

[0048] Various means can be used to determine the origin of the sound; for example, 
in some simple installations it can be sufficient to use two microphones. Also four 
microphones could be used. Three microphones only determine position of the hand in 
two, say x and y dimensions. Two cameras are able to determine all three dimensions, 
the x and y axes being given by both cameras, and the z-axis is deriving from the 
disparity map computed by combining simultaneous pictures acquired by the camera pair. 
With three microphones, redundancy is obtained between x and y axes across sound and 
pictures, which enables the system to check that the hand is where the approved sound is 
coming from. A fourth microphone mounted at a wall of the room (not the same as the 
display) would enable the cross-check of valid sound command of hand position across 
all three dimensions. 

[0049] Having thus described at least one illustrative embodiment of the invention, 
various alterations modifications and improvements will readily occur to those skilled in 
the art. Such alteration, modification, and improvements are intended to be within the 
spirit and scope of the invention. Accordingly, the foregoing description is by way of 
example only and is not intended to be limiting. The invention is limited only as defined 
in the following claims and the equivalent thereto. 
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